System and method for adapting auditory biofeedback cues and gait analysis using wireless signals and digital responses

ABSTRACT

A method for adapting auditory biofeedback cues to adjust a user&#39;s gait includes receiving a series of respective first signals from each sensor of an integrated sensor system and converting the series of respective first signals into a series of respective second signals. Each respective second signal within the series of respective second signals can be quantized as an audio biofeedback cue and modified so the timing of each respective second signal is aligned to a pre-selected temporal or musical grid. The user&#39;s gait is then analyzed as a function of the audio biofeedback cues and the pre-selected temporal or musical grid can be adapted to adjust entrainment of the user&#39;s gait.

FIELD OF THE INVENTION

The present invention generally relates to a system and method forgenerating wireless signals generated from physical movement sensorsand/or similar devices coupled to a person's body. The invention moreparticularly relates to such a system and method which also enables theselective control of the digital responses to the generated signals,including MIDI data (Musical Instrument Digital Interface), sounds,visuals, and/or interactive responses, for example. Still moreparticularly, the present invention relates to a system and method foradapting auditory biofeedback cues and gait analysis using wirelesssignals and digital responses.

BACKGROUND OF THE INVENTION

Electronic auditory and/or visual input/output systems and componentssuch as musical controllers for MIDI compatible equipment, electronictap dancing shoes, and the use of touch-screen interfaces for the remotecontrol of electronics are well known in today's digital world. Musicalor MIDI controllers are the hardware interface for the control ofdigital musical instruments. MIDI (Musical Instrument Digital Interface)is an industry standard data protocol for digital instrumentsestablished in the 1980's that remains in use to the present day. Amusical keyboard is most typically used to “control” sound banks orsynthesizers that are wired to the controller with MIDI cables.Percussive MIDI controllers, such as the Roland Corporation's OCTAPAD®,contain large rubber pads that when hit, trigger digital samples. MIDIcontrollers may also contain sliders, knobs or buttons that controlrecording of MIDI music. Novation's Launchpad uses buttons to act asswitches for recording, or as drum pads for performing music.

Alternative MIDI musical controllers produced and marketed in the pasthave included breath controllers offered by Yamaha as optionalaccessories for their line of keyboard synthesizers produced in the1980s (the DX7, DX11, CS01, and others). These breath controllersallowed the use of breath pressure to have the synthesizer sendcorresponding MIDI continuous control messages to modify the soundoutput. In the wake Yamaha's controller, other manufacturers have madeand offered breath controllers that are free-standing and allow the userto add breath control of MIDI continuous controller messages toinstruments lacking that control as a built-in feature. For example theTEControl USB MIDI Breath Controller can be used with a wide range ofMIDI compatible musical equipment or computer software that accepts MIDImessages.

Previous inventors have tried to develop electronic tap-dance shoes thatuse pressure sensors or other means to detect a dancer's activity andthen send corresponding MIDI notes, either through cables or wirelessly.For example, U.S. Pat. No. 5,765,300 entitled Shoe Activated SoundSynthesizer Device is directed to a shoe activated sound synthesizerdevice that enables movement of a shoe to be translated into audiblesounds. The sound synthesizer device consists of a shoe in which thereis disposed at least one trigger element capable of producing a triggersignal when the shoe is flexed to a predetermined degree. As the shoe isworn and is brought into contact with the floor, the shoe is flexed. Bybringing different parts of the shoe into contact with the floor in acontrolled manner, a person can selectively control the production oftrigger signals from any trigger element contained within the shoe. Asound synthesizer circuit is provided that is coupled to each triggerelement contained within the shoe. The sound synthesizer circuitproduces an audible sound, via a speaker, when a trigger signal isreceived from the shoe. Pressure sensors have also been embedded infloors or floor-mounted surfaces and used as arcade or home gamecontrollers: examples include Nintendo's Wii Fit Balance Board andKonami's Dance Dance Revolution. The Nike+ FUELBAND® is a device thattracks a user's movement and activity in order to track progress infitness training. The Nike+ sensor may transmit a data packet to areceiver directly attached to a mobile device.

Additionally, wireless remote control of electronics hardware through anapplication on a mobile device or tablet computer is an expanding field.One example is the GoPro App, which allows a user full remote controlover a GoPro camera's functions and record button, as well as providinga useful preview image of what the camera is photographing, if forexample it is attached to the top of a helmet deeming the viewfinder notvisible.

While the above prior art provides examples of signal generation throughphysical movement, there remains a need for a system and method whichallows the manipulation of the response signals in real-time. Thepresent invention addresses this, and other, needs in the art.

BRIEF SUMMARY OF THE INVENTION

A system and method for generating wireless signals from the physicalmovement of a person utilizing a movement detection mechanism attachedto the person wherein the system allows a person to manipulate thegenerated wireless signals to selectively control digital responseswhich may be in the form of sensory-perceivable outputs such as soundsand/or visual effects, for example, through the person's physicalmovement. Movement sensors attached to the person (e.g., on one foot orboth feet) communicate with other system components such asmicroprocessors, transceivers and/or tactile interface controls, forexample, to wirelessly send signal pulses from a person to a computer ormobile device and allow the person wearing the sensors and/or anotherperson to selectively control the dynamics of the digital responses soto create a unique sensory output.

In accordance with an aspect of the present invention, a system forcreating a sensory output from a user's physical movements comprises oneor more sensors configured to be removably attachable to a user's body.The one or more sensors may be adapted to detect movement and trigger asignal containing movement data in real-time. A transceiver is operablycoupled to the one or more sensors to transmit the real-time signal anda receiver is coupled to a computing device with the receiver configuredto receive the transmitted real-time signal. The computing deviceconverts the movement data to an output signal wherein the output signalmanifests as a sensory output comprising one or more of a visual signal,an interactive effect signal, a Musical Instrument Digital Interface(MIDI) signal or an audio signal.

In a further aspect of the present invention, the system may furthercomprise a tactile interface unit coupled to the receiver and computingdevice wherein the tactile interface unit is operable to selectivelycontrol and manipulate the output signal. The tactile interface unit maybe configured to be removably attached to the user's body and to beoperable by the user. Moreover, the tactile interface unit, the receiverand the computing device may be housed within a single mobile deviceconfigured to be removably attached to the user's body. Alternatively oradditionally, the tactile interface may be remotely located from theuser and may be operated by a second party.

In another aspect of the present invention, the tactile interface unitis an application running on a mobile device and is configured forwireless remote control of the output signals produced by the computingdevice. The output signals may be characterized by a note's length, aMIDI channel used, a MIDI continuous controller number sent, or anyother parameters that can be modified in the digital output of theaudio, visual or interactive effect signals.

In still a further aspect of the present invention, the transceiver maybe a radio transceiver configured to generate and transmit wirelesspre-MIDI signals, wherein the pre-MIDI data signals from the transceiverare converted by the computing device into MIDI notes, MIDI continuouscontroller messages or similar data protocol. The computing device mayconvert the wireless pre-MIDI signals into actual MIDI data at a higherdata rate than the MIDI protocol to thereby reduce latency in the outputof the output signal.

In an additional aspect of the present invention, at least one of theone or more sensors resides in a shoe configured to be worn by the user.A shim may also be placed within the shoe wherein the shim is configuredto position the at least one sensor at a transverse arch or heel of theuser's foot.

In another aspect of the present invention, the transceiver is housedwithin a transceiver box where the transceiver box further includes amicroprocessor programmed to include a peak-detection algorithm. Themicroprocessor utilizes the peak-detection algorithm to convert movementdata generated by the one or more sensors into discrete digital signalsindicating event onsets to be transmitted by the transceiver.

In a further aspect of the present invention, one or both of thereal-time signal and the output signal are communicated wirelessly overa local area network (LAN), a wide area network (WAN), a Cloud or theinternet.

In still another aspect of the present invention, a system for creatingsensory outputs from physical movements of a plurality of userscomprises a respective set of one or more sensors configured to beremovably attachable to a respective user's body. The one or moresensors may be adapted to detect movement and trigger a signalcontaining movement data in real-time for the respective user. Arespective transceiver may be operably coupled to each respective set ofone or more sensors to transmit the real-time signal for the respectiveuser. A receiver may be coupled to a computing device and be configuredto receive the transmitted real-time signal from each user. Thecomputing device may then convert the movement data from each user to aseries of output signals wherein the output signals manifest as sensoryoutputs comprising one or more of a visual signal, an interactive effectsignal, a Musical Instrument Digital Interface (MIDI) signal or an audiosignal.

In a further aspect of the present invention, a tactile interface unitmay be coupled to the receiver and computing device where the tactileinterface unit is operable to selectively control and manipulate theoutput signals. Additionally or alternatively, a respective tactileinterface unit may be configured to be removably attached to eachrespective user where each respective tactile interface unit may be incommunication with the receiver and computing device wherein eachrespective tactile interface unit is operable to selectively control andmanipulate the output signals.

BRIEF DESCRIPTION OF THE DRAWINGS

The above-mentioned and other features and advantages of this invention,and the manner of attaining them, will become apparent and be betterunderstood by reference to the following description of the invention inconjunction with the accompanying drawing, wherein:

FIG. 1 is a schematic view of a system for generating wireless signalsfrom physical movement in accordance with an embodiment of the presentinvention;

FIG. 1A are schematic views of alternative tactile interfaces that maybe used within a system of the present invention;

FIG. 2 is a schematic view of a system for generating wireless signalsfrom physical movement in accordance with another embodiment of thepresent invention;

FIG. 3 is a schematic view of a device used to generate wireless signalsfrom physical movement in the embodiments of the system shown in FIGS. 1and 2;

FIG. 4 is a schematic view of a pressure sensor layout used within thedevice shown in FIG. 3;

FIG. 5 is a schematic view of a signal transceiver bracket used withinthe device shown in FIG. 3;

FIG. 6 is a schematic view of a signal transceiver strap used within thedevice shown in FIG. 3;

FIG. 7 is a schematic view of the components within a signal transceiverused within the system shown in FIGS. 1 and 2;

FIG. 8 is a schematic view of an alternative device used to generatewireless signals from physical movement in the embodiments of the systemshown in FIGS. 1 and 2;

FIG. 9 is a schematic view of an alternative pressure sensor layout usedwithin the device shown in FIG. 3;

FIG. 10 is a schematic view of a system for generating wireless signalsfrom physical movement by more than one performer in accordance withanother embodiment of the present invention;

FIG. 11 is a schematic view of a system for generating wireless signalsfrom physical movement by more than one performer wherein each performermay remotely control system outputs in accordance with anotherembodiment of the present invention;

FIG. 12 is a schematic view of a system for generating wireless signalsfrom physical movement by more than one performer wherein anon-performer may remotely control system outputs in accordance withanother embodiment of the present invention;

FIG. 13 is a plot, along with associated visual aid, showing pressuresensor recordings when analyzing a user's gait in accordance with anaspect of the present invention;

FIG. 13A is a side perspective view of an embodiment of an alternativesensor unit in accordance with a further aspect of the presentinvention;

FIG. 13B is a bottom view of the alternative sensor unit shown in FIG.13A;

FIG. 14 is the plot shown n FIG. 13 indicating exemplary diagnosticcriteria during a gait analysis;

FIG. 15A is an auxiliary plot of an exemplary left heel pre-MIDI outputfor the plot shown in FIGS. 13 and 14;

FIG. 15B is an auxiliary plot of an exemplary left toe pre-MIDI outputfor the plot shown in FIGS. 13 and 14;

FIG. 16A is a diagrammatic plot of the quantization and output of MIDInotes for the plot shown in FIG. 15A;

FIG. 16B is a diagrammatic plot of the quantization and output of MIDInotes for the plot shown in FIG. 15B;

FIG. 17 is a diagrammatic view of a system for adapting auditorybiofeedback cues to adjust a user's gait; and

FIG. 18 is a flow chart of an exemplary method for adapting auditorybiofeedback cues to adjust a user's gait in accordance with an aspect ofthe present invention.

Similar reference characters refer to similar parts throughout theseveral views of the drawings.

DETAILED DESCRIPTION OF THE INVENTION

In operation, one or more sensors (such as, but not limited to, pressuresensors, accelerometers and the like) may be placed on or near a foot orboth feet. In the act of walking, running, dancing or other movement, asensor may be triggered and read by detector, such as an analog-todigital converter. A microprocessor may be attached to the sensor andmay transmit a wireless signal pulse to a computer. Computer softwareresident on the computer may then convert the wireless signal pulsesinto MIDI data (or a similar data protocol) that may be recorded asinteroperable data or may be assigned to digital responses such as, butnot limited to audible sounds including musical notes and beats; visualfeedback in lighting effects or digital graphics; or interactiveresponses from a video game or digital display.

A user may record, loop or modify the MIDI data or the dynamics of thedigital responses in real-time by using a suitable interface, such asthrough tactile finger movements upon an interface in conjunction withmovement of his or her legs and feet. By way of example, the dynamics ofthe responses that may be changed in real-time by coordinating fingerand feet movements include, but are not limited to, the modification ofthe precise timing and length of the digital effects produced or thequalities of the visuals or sounds that are being generated by aperson's physical movements. As used herein, the term real-time refersto digital signals or responses that occur perceivably instantaneouswith the underlying physical movement. As a result, systems design inaccordance with the teachings of the present invention may haveapplications as a musical and/or scientific instrument used in suchactivities as dance performance, music production, athletic activities,art projects, entertainment systems, health diagnostics or medicaltherapies and the like.

Turning now to the drawings, with particular reference to FIG. 1thereof, a system for generating wireless signals generated fromphysical movement sensors is generally indicated by reference numeral100. System 100 may comprise include a signal generation component 102configured to wirelessly communicate movement related sensor data to asignal receiver component 104 before eventual broadcast (i.e. audioand/or visual responses) via one or more output modalities 106. Asdiscussed in greater detail below, signal generation component 102 maybe worn upon a user 108 and include one or more sensor and radiotransceiver units 110 located on or proximate to the user's shoe or foot112 and, optionally, a tactile interface unit 126. Respective sensor andradio transceiver units 110 may be worn on one or both feet. Signals 115generated by the sensors and transmitted by the transceivers may bereceived by a radio receiver 114 in communication with a computingdevice 116, such as a smart phone, laptop, tablet or PC computer.Software resident within computing device 116 may then condition thereceived signals before eventual output 117 to appropriate outputdevices, such as via VGA output 118 (for visual signals), HDMI output120 (for interactive effects), MIDI output 122 (for digital notes andbeats) and/or digital output 124 (for sounds). A tactile interface unit126 may also be coupled 127 to receiver component 104 wherein tactileinterface 126 may be used for control and manipulation of output 117. Asshown in FIG. 1A, non-limiting examples of possible tactile interfaceunits 126 may include a touch-screen device 126A attached to the wrist,an interface responsive to an app on a mobile device 126B held in thehand or a MIDI controller keyboard 126C manipulated by the fingers.

As shown generally in FIG. 2, an alternative embodiment of a system 130for generating wireless signals generated from physical movement sensorsmay utilize a single mobile device 126B (such as a smart phone or tabletcomputer) that contains a built-in touch-screen tactile interface 126and receiver component 104 (radio receiver 114 and computing device 116)configured for digital output of sound to headphones 134. Mobile device126B may wirelessly receive signals 115 generated by sensors and radiotransceiver units 110. In this manner, user 108 may use system 130 as apersonal musical instrument capable of producing controllable digitalsounds by virtue of coordinating commands of the tactile interface withphysical movements of the feet and body.

In the embodiments shown in FIGS. 1 and 2, tactile interface unit 126may be strapped to the wrist (FIG. 1) or held in the hand (FIG. 2). Bypressing the interface with the fingers, user 108 may wirelessly changethe precise timing and/or length of the digital effects and/or thequality of the digital responses produced, including the rhythmic timingand/or length of MIDI notes, for example. Via tactile interface unit126, user 108 may be able to change the assignment of a particular MIDInote to a particular sensor and/or a particular physical movement thatis activated by a pressure sensor and/or inertial measurement unit. Inthis manner, MIDI continuous controller data can be modified to createdynamic changes in digital effects and/or user 108 launch presets ofdifferent combinations of digital effects in order to transpose orarpeggiate musical notes and/or animate visual patterns. Tactileinterface unit 126 may also display visual feedback confirming thedigital effects being produced, such as a preview of graphics that arebeing projected on a larger screen and/or the user's current score ifthe system is being used in a multi-user video game environment, forexample. As a result the conclusive aesthetic result of system 100/130is a series of coordinated digital responses in the form of soundsand/or visuals that are triggered and controlled by a user's physicalmovements.

With reference to FIG. 3, an embodiment of a sensor and radiotransceiver unit 110A is shown. Sensor and radio transceiver unit 110Amay be configured to mount to a user's shoe 112 a with pressure sensors114 positioned beneath the user's foot 113 and adhered to the innersole142 of shoe 112A. With additional reference to FIG. 4, utilizing footmovement to create wireless signals 115 does not require specializedfootwear. For instance, one or more pressure sensors 140 may be attachedto a shim or raised support 144 that is positioned onto a removableinnersole 142 of any suitable shoe. In accordance with an aspect of thepresent invention, at least one pressure sensor 140 (an optional shim144) is positioned between the transverse arch 146 of foot 113 andinnersole 142 when inserting foot 113 into shoe 112A, and moreparticularly in the area of the transverse arch of the foot locatedbetween the ball of the foot 145 and the smallest toe 147. A pressuresensor 140 (an optional shim 144) may also be placed near the heel 149of foot 113. The size and orientation of each shim 144 is selected so asto ensure contact between the innersole and the foot, while alsominimizing user awareness of shim 144 and/or pressure sensor 140 and anydiscomfort that may result therefrom. Further, the shape and size ofpressure sensors may be modified or increased/decreased in order toselectively define the zone of sensitivity, that is, where on the footis pressure required to be sensed by the sensor 140 before initiating asignal 115.

As shown in FIG. 3, a radio transceiver box 148 may be releasablysecured to laces 150 of shoe 112A. For instance, as shown in FIG. 5,radio transceiver box 148 may be secured to laces 150 by way of abracket 152 slid under laces 150 of shoe 112A. Radio transceiver box 148may then be releasably mounted to bracket 152 via a releasable fastener(not shown), such as a snap, magnets, hook-and-loop material and thelike. In this manner, radio transceiver boxes 148 may be removed whennot needed and/or may be shared between people. Alternatively, as shownin FIG. 6, a strap 154 may be wrapped around the body of the shoe withthe radio transceiver box 148 releasable attached to the strap.

As generally shown in FIG. 7, radio transceiver box 148 may include ahousing 156 containing a printed circuit board 158 having an analog todigital converter circuit 160 configured to receive analog sensor datafrom pressure sensors 140 (such as via external jack 162) and convertsuch analog sensor data into digital signals for interrogation bymicroprocessor 164. Interrogated digital signals may then be wirelesslytransmitted via wireless transceiver 166 (which may also include anantenna 168 configured for digital broadcast). In accordance with anaspect of the present invention, radio transceiver box 148 may alsoinclude an inertial measurement unit 170 configured to sense and outputsensor data regarding movement of radio transceiver box 148. Withoutlimitation thereto, inertial measurement unit 170 may include one ormore of an accelerometer, gyroscope and a magnetometer. By way ofexample, signals outputted by inertial measurement unit 170 may triggera selected output 117 solely through user movement without requiringfootfall and activation of pressure sensor 140. Radio transceiver box148 may also include a battery 172 configured to provide necessary powerto box components. It should be understood by those skilled in the artthat any suitable battery may be used, including but not limited tonon-rechargeable and rechargeable batteries. Light emitting diodes(LEDs) 171 may be also included to provide visual indication that radiotransceiver box 148 and its various internal components are operatingproperly or to display colors synchronized to musical notes or from theuser's settings. It should also be noted that by miniaturizing theelectronics of the of the radio transceiver unit 110A, the scale, weightand power consumption may be reduced.

Turning now to FIG. 8, a hard soled dance shoe 112B may be modified toaccommodate one or more sensors 140 and a radio transceiver box 148. Abattery 172 (and optional battery recharging port 174 in the case ofbattery 172 being a rechargeable battery) may be embedded into the heel176 of the shoe. By situating battery 172 within heel 176, the size andweight of radio transceiver box 148 may be reduced so that radiotransceiver box 148 may more comfortably be attached to a strap 178 ondance shoe 112B.

As shown in FIG. 9, in an alternative embodiment 110B, one or morepressure sensors 140 may be directly affixed to a user's foot 113, suchas through an adhesive 180. A radio transceiver box 148 may then bereleasably secured to the user's body, such as at or near user's ankle182. In this manner, user 108 may generate wireless signals 115 withoutrequiring any shoes, but merely through impact of his or her bare footupon a surface. While not limited strictly thereto, embodiment 110B maybe suitable for use as a scientific instrument for developing physicaltherapies to improve a person's foot placement and/or gate, includingtherapies for foot pronation, walking disorders and/or physical movementdisabilities, for example. By adhering pressure sensors 140 directly tothe skin of the foot and strapping radio transceiver box 148 to ankle182, a user can walk barefooted while the system would trigger sounds toencourage proper heel-to-toe foot movement and/or provide interactiveresponses and/or the recording of data, for example, to assist theperson in therapy or diagnosis. Alternatively, user 108 may elect towear shoes while pressure sensors 140 are directly affixed to foot 113.

FIGS. 10 through 12 show alternative embodiments of the presentinvention wherein a respective system generates wireless signalsgenerated from physical movements from a plurality of individuals. Byway of example, such individuals may be members of a dance/musicensemble or may be players in a multi-user video game environment.

With reference to FIG. 10, a multi-person system 200 has a group ofpeople 208 wherein each member 208A-208D of the group is equipped with arespective sensor and radio transceiver unit 210A-210D. It should benoted that while shown and described as having four members,multi-person system 200 may be used with any size group of users andsuch alternative group sizes are to be considered within the teachingsof the present invention. Each respective sensor and radio transceiverunit 210A-210D may be in wireless communication with common signalreceiver component 204, such as but not limited to a turnkey computer216 having an external or internal radio receiver 214. Signal receivercomponent 204 may then be operated by a dedicated DJ or technician 217,such as via a MIDI controller 226, to produce sound, lighting or videoeffects, such as via speaker/lighting unit 228. In this manner, group208 may perform as a dance or musical ensemble or may interact with adance simulation gaming program or app (such as country-line dancing)that responds to each member's physical movement without confining theindividual members to a camera-based or environmental motion trackingsystem.

Multi-person system 300, as shown in FIG. 11, is similar to multi-personsystem 200 described above, with the exception that DJ/technician 217may be omitted. Rather each member 308A-308D of the group includes arespective sensor and radio transceiver unit 310A-310D wirelesslycoupled to a respective tactile interface unit 326A-326D which in turnis in wireless communication with a common signal receiver component 304on a local area network (LAN) that can be controlled remotely by eachmember 308A-308D via each respective tactile interface unit 326A-326D.Signal receiver component 304 may output sound, lighting or videoeffects signals similar to system 200 or signal receiver component 304may comprise an external or internal radio receiver 314 coupled to aturnkey computer 316 having built in speakers 328 and video display 329.In this manner, and by way of example, the group may perform as a danceor music ensemble or players in a multi-user video game.

FIG. 12 shows a multi-person system 400 similar to system 300 describedabove wherein individual members 408A-408E of a group includes arespective sensor and radio transceiver unit 410A-410E wirelesslycoupled to a respective tactile interface unit 426A-426E which in turnis in wireless communication with the Internet, Cloud or a wide areanetwork (WAN) 427. Each respective sensor and radio transceiver unit410A-410E and its digital output may be controlled remotely by eachmember 408A-408E via each respective tactile interface unit 426A-426E.Audio outputs may be heard by each member via respective headphones434A-434F Additionally or alternatively, a dedicated DJ or technician408F (who may also be an active member of the group and have arespective sensor and radio transceiver unit 410F and respective tactileinterface unit 426F) may control signal outputs, such as through a MIDIcontroller 426G similar to that describe above with regard to FIG. 10.In this manner, system 400 may enable members to work collaboratively inreal-time even when one or more of the members is remotely located fromthe other members of the group. By way of example, system 400 may enableconcerted group activities in multi-user video games or assisting a teamto move together in a drill or routine.

In each of the above embodiments, when a user steps onto a pressuresensor 140, it is activated. When a user spins the body or moves a legthrough the air, inertial measurement unit 170 is activated. Bothpressure sensor(s) 140 and inertial measurement unit(s) 170 sendelectrical signals to the analog-to-digital converter circuit 160. (SeeFIG. 7). Microprocessor 164 detects a peak in the pressure wave oracceleration curve and determines a discrete point that is transmittedas a wireless pulse in the signal from the transceiver 166 to thereceiver 114. Computer 116 (or tactile interface unit 126) may thenconvert the wireless signal of pulses into audible sounds includingmusical notes and beats and/or visual feedback in lighting effectsand/or digital graphics and/or interactive responses from a video gameor digital display, for example. Microprocessor 164 may be programmed toinclude a peak-detecting algorithm that converts these continuouspressure waves and acceleration curves to discrete pulses. However, inoperation, the user may require a single note to be generated from asingle footstep, so any additional pulses are be filtered out by a clockfunction within the software to thereby convey a response of a singlenote from a single footstep. This could be optionally activated in thesystem settings. In typical physical movements, such as walking, peoplealternate footsteps between the left and right feet. In order togenerate a scale of musical notes from these alternating footsteps, eachsignal pulse emanating from the combined left and right radiotransceivers is numbered sequentially. The receiver and computer mayalso contain software capable of numbering the sequence of pulses, andtranslating them into MIDI note numbers or pitches, beats or tones in amusical scale.

In each of the above embodiments, when movement is detected, such asthrough pressure sensor 140 and/or inertial measurement unit 170, awireless signal 115 of pulses is transmitted via wireless transceiver166 to receiver component 104. By coordinating inputs within tactileinterface 126 with physical movement of the feet and body, a person canmodify the timing and length of the digital effects and/or the qualitiesof the visual and/or sound responses that are being generated by aperson's physical movements.

In accordance with an aspect of the present invention, signal processingflow begins with pressure waves and acceleration curves generated by oneor more sensors 140/inertial measurement units 170 attached to the bodyof a user 102 that are triggered by physical movement. (See e.g., FIGS.1 and 7). A peak-detecting algorithm stored on the microprocessor 164identifies the peak of the pressure wave or acceleration curve andsignals it as a low latency wireless pulse to the radio receiver 114.Computer 116 (or tactile interface unit 126) converts the signal ofpulses into operable data, for example MIDI notes and beats. Thecomputer software numbers the pulses in a sequence determined by theuser, then filters out unwanted pulses, quantizes the notes to a musicalgrid or modifies them based on the timing of the measures of music bysending the notes through an envelope. The resulting digital responsecan be seen or heard in real-time, and for example MIDI data can berecorded by the computer or controlled and manipulated in real-time by atactile interface unit either worn by the person generating the physicalmovement (see FIG. 2) or by a DJ/Technician (see FIG. 10) controllingthe dynamics of multiple people wearing sensors and radio transceivers.By way of example, a user or DJ/technician may modulate all of the notesbeing generated by multiple users to a different musical scale, chord orkey at a specific instant in time or produce a unified aesthetic changein the digital output of the system.

As used herein, the term “real-time” refers to digital signals orresponses that occur perceivably instantaneous with the underlyingphysical movement. That is, the digital signal or response originates atthe sensor or inertial measurement unit within about 500 milliseconds toabout 50 nanoseconds, and more preferably between about 1 millisecondand about 100 milliseconds, from the time of peak detection by thealgorithm to transmission of the digital signal to the receiver. In thismanner, an output signal and the resultant audio, visual and/or othereffects are perceived by the user and any spectators as occurringsubstantially simultaneously with the user's movements. In other words,there is no noticeable delay between a user's movements and theresultant effect.

A method for synchronizing the MIDI signals of the system may beutilized for generating harmonious and rhythmically unified digitalresponses from a group of persons (see FIGS. 10-12) generating pulsesfrom physical movement. The synchronization may involve quantizing theinput of the MIDI notes to the beats and measures of the music. Thisinvolves modifying the precise time notes are played by shifting them toan established temporal grid. The use of a pitch envelope can modifyincoming notes (generated by the feet) to trigger specific pitches atset beats of music or instants in time. The modifications of MIDIsignals may either expand or limit the amount of pitches generated bythe feet.

The synchronization of signals generated by multiple users to a singlereceiver and computer produces the digital response of dancing with apartner, or choreographed group movements. Synchronization of digitalresponses transmitted through a LAN computer network (see FIG. 11) orthe internet (see FIG. 12) enables remote users to dance together, orsend signals from physical movements generated by multiple users acrosscomputer networks in real-time.

As shown in the above reference embodiments, a computer may include aninternal radio receiving unit or an external receiving unit coupled to acomputer, such as via a USB port. In accordance with an aspect of thepresent invention, the signal receiver component 104 may utilize theIEEE 802.15.4 networking protocol for fast point-to-multipoint orpeer-to-peer networking. Bluetooth LE (low energy) and/or Bluetooth 4.0and later revisions of the protocol may offer a fast data transferinterval for low latency, real-time, wireless signal transmission andreception. Utilizing the Bluetooth 4.0 protocol may allow radiotransceiver boxes that are removably attachable to a person's body tocommunicate directly with Bluetooth 4.0 supported devices.

Tactile interface unit 126 may provide for real-time control of thedynamics of the digital response effects emanating from the computer.The tactile interface may be operated by a medical clinician or the userthemselves and may be capable of modifying the MIDI signals produced bythe radio receiver and computer. In operation, a user moves his or herfeet thereby activating sensors and generating wireless pulses asdescribed above. Coordinately, the user's fingers touch tactileinterface unit 126 so as to selectively modify the note pitch, length orany other MIDI parameters.

As described above, tactile interface unit 126 may be a touch-screendevice attached to the wrist, be contained in an app on a mobile deviceheld in the hands, or be a MIDI controller keyboard manipulated by thefingers. Tactile interface unit may be utilized to alter the systemsettings such as adjusting the sensitivity of the pressure sensors,changing the preset sound or digital effect in real-time or altering thepitch of the note in real-time. Any digital event onset including MIDIevents can be triggered as a digital output from the computer anddynamics such as the length and type of sound or visual effect can becontrolled and modified by the tactile interface.

Turning now to FIGS. 13-18, and with particular reference to FIG. 13, anexemplary gait analysis study using an embodiment of system 100,described above, is shown. As seen in the top portion of FIG. 13, a user500 has a first system 502 (analogous to system 100 described above andas will be discussed in greater detail below) mounted about the foot504. By way of example and without limitation thereto, first system 502may be mounted directly onto foot 504 or may be mounted within shoe 506or may be attached to the external surface of shoe 506, such as via astrap 507. In any event, first system 502 is configured to include aheel sensor 508 adapted to detect heel pressure and a toe sensor 510adapted to detect toe pressure, and may also include one or moreoptional inertial measurement units (not shown, such as inertialmeasurement unit 170 as seen in FIG. 7). With reference to plot 512 ofFIG. 13, first system 502 may be worn on the left foot as describedabove. With additional reference to FIGS. 15A, 16A and 17, a secondsystem 514 may be worn on the right foot and include a respective heelsensor 515 and toe sensor 517.

While shown and described as distinct heel/toe sensor units 508/510(left foot) and 515/517 (right foot), it should be understood by thoseskilled in the art that one or both heel/toe sensor units may beincorporated within a single, whole-foot sensor 502 a mounted onto orwithin shoe 506 a (FIGS. 13A and 13B). In accordance with this aspect ofthe present invention, whole-foot sensor 502 a may include a pluralityof individually triggered sensor elements 504 a that may be arranged asa grid or array 506 a. As a wearer steps, only those sensor elements 504a which are impacted will produce a corresponding signal. Thus,whole-foot sensor 502 a is configured to generate a series of sensordata for each sensor element 504 a across the whole of array 506 a.

The wearer, physical therapist, doctor, clinician or other third partymay selectively isolate (such as via microprocessor 164 or computingdevice 116, described above) specific sensor elements, such as thosethat define heel region 508 a (denoted by the letter Y in FIG. 13B) andtoe region 510 a (denoted by the letter X in FIG. 13B). Heel region 508a and toe region 510 a may then functionally operate analogous to heelsensor 508 and toe sensor 510, as described above and further discussedbelow. Thus, for the sake of clarity, the following discussion willrefer to heel sensor 508 and toe sensor 510, although it should beunderstood that such teachings may equally apply to heel region 508 aand toe region 510 a.

As shown in plot 512, both heel sensor 508 and toe sensor 510 may beactivated as user 500 takes a step such that microprocessor (e.g.microprocessor 164 on radio transceiver unit 110, FIGS. 1 and 7) ofsystem 502 generates a respective pre-MIDI pressure curve 518, 520 as afunction of time for heel and toe sensors 508, 510. As can be seen inFIG. 13, sensors 508 and 510 measure contact pressure (e.g., heelstrikes 522) and release (e.g., toe off 524) of the user's heels andtoes while the user steps. With additional reference to FIG. 14,respective pre-MIDI pressure curves 518 (left heel), 520 (left toe) mayoffer diagnostic information regarding the gait of user 500.

By way of example, the length of time of the user's stride time may bemeasured as the time difference between successive heel strikes 532 a,532 b, while the user's swing time (i.e., the length of time the foot isswinging through the air) may be measured as the time difference betweentoe off 534 and heel strike 532 b and the stance time (i.e., the lengthof time the foot is in contact with the floor) may be measured as thetime difference between heel strike 532 a and toe off 534. This data mayalso be coupled with inertial measurement unit data, such as thatreceived from inertial measurement unit 170, so as to enable measurementof the stride length or step distance, along with other step/gaitperformance characteristics.

In addition to the above, the integrated pressure sensors 508/510 andinertial measurement unit 170 may be able to detect user state; that is,whether the wearer is sitting, standing or moving (e.g., walking,running, hopping etc.). By way of example, user state data may beimportant during a physical therapy (PT) session, such as for a patientwith Parkinson's disease. For instance, a frequent PT exercise involveswalking a short distance. User state data may indicate step frequency,stride length, whether the patient has stopped to rest or sit down, orwhether the patient is experiencing “frozen feet” which is common forpatients with Parkinson's disease. System 502 also provides instantauditory biofeedback to encourage gait improvements during the PTsession.

As shown in FIGS. 15A and 15B, and 16A and 16B, a computing device, suchas microprocessor 164 or computing device 116 (FIGS. 1 and 7), includesa computer processor that is configured to detect the edge and/or peakof the pre-MIDI data. In accordance with an aspect of the presentinvention, the computer processor may execute an event onset algorithmto detect event onset of the MIDI notes from the pre-MIDI data. As willbe described in greater detail below, the event onset algorithm mayinterrogate the pre-MIDI data to detect event onset through one or bothof peak detection or edge detection, and following edge (or peak)detection, microprocessor 164 or computing device 116 may then quantizethe pre-MIDI data to generate respective MIDI signals which willultimately comprise MIDI notes or MIDI continuous controller messages.

With specific reference to FIGS. 15A and 15B, when a user placespressure (a stepping force) upon, such as, heel sensor 508 of left footsystem 502, a pre-MIDI pressure curve 528 is generated. Similarly, astepping force on toe sensor 510 of left foot system 502 generatespre-MIDI pressure curve 530. It should also be noted that foot motiondata may be generated via inertial measurement unit 170. The pressureand inertial measurement unit data may be multiplexed so thatinformation of multiple events can be contained within the same signal.The event onset algorithm, via microprocessor 164 or computing device116 processor, may then interrogate each pre-MIDI pressure curve 528,530 (and, optionally, the inertial measurement unit data) to detect anevent onset whereby the stepping force meets and exceeds a pre-selectedforce threshold 536, 538 respectively, at which time the event onsetalgorithm initiates quantization of respective pressure curves 528, 530to define left heel MIDI signal 540 and left toe MIDI signal 542.Quantization of pressure curves 528, 530 continues until the appliedforce to sensors 515, 517 falls below a respective pre-selected forcethreshold which may be the same or different that respective thresholds536, 538.

Respective MIDI signals 540, 542 define a respective MIDI note orcontroller message corresponding to the time domain when thecorresponding sensor 508, 510 was actuated with a force greater than therespective pre-selected force thresholds 536, 538. Each pre-MIDIpressure curve 528, 530 continues to be interrogated by the event onsetalgorithm until the force applied to the sensor(s) 508, 510 againtransitions past its pre-selected force threshold 536, 538, at whichpoint the next successive MIDI signals on the left foot 544 (left heel)and 546 (left toe) are detected and quantized. FIG. 15B shows similarMIDI signals 548, 550 following detection and quantization of pre-MIDIpressure curves 554, 552 of the right heel and toe sensors 515 and 517,respectively, once the respective stepping force exceeds pre-selectedforce thresholds 548, 550, respectively. It should also be noted thatthe sensitivity of system 500 may be selectively adjusted by definingthe event onset (peak and/or edge detection) thresholds for each sensor508, 510, 515, 517. Thus, the threshold can be set so that the eventonset algorithm accurately triggers sound with the user'sperception/feeling of heel and toe touches, thereby providing a “sensoryresponse” as biofeedback.

With reference to FIGS. 16A and 16B, in accordance with a further aspectof the present invention, once pre-MIDI signals have been quantized toMIDI notes or controller messages, computing device 116 may calibratethe MIDI notes as auditory biofeedback cues. Initially, a usersequentially places maximum weight-bearing pressure on each sensor. Agait analytic algorithm then sets an output value of the maximumweight-bearing pressure at 100%. MIDI signals 540, 542, 544, 546, 548,550 may then be outputted as respective square waves 560, 562, 564, 566,568, 570 wherein the magnitude of each square wave is calculated as afunction of the maximum weight-bearing pressure and displayed as arespective percentage of that pressure. MIDI signals 540, 542, 544, 546,548, 550 may then be ultimately realized as respective musical notes orother sounds.

The volume and pitch of each respective outputted note or sound may becorrelated to the weight percentage of the MIDI signal square wave so asto provide auditory biofeedback that alerts the user, such as if theyare favoring the left or right limb while walking. Furthermore, bycalibrating the heel and toe touches with the length of the stance timealong the X-axis, the system can produce biofeedback that matches eachstep. Thus, system 500 may efficiently count the alternating left andright steps, 572, 574, 576. The accuracy of the step counting may beimproved by utilizing the auditory biofeedback to calibrate the systemas described above—that is, by stepping and “tuning” the sensitivity ofthe auditory sensory response (adjusting the pre-determined thresholdfor each sensor) until it matches the physical movement of foot toucheswhile walking.

In accordance with an aspect of the present invention and as showngenerally in FIG. 17, the timing of the auditory biofeedback cues may bemodified and arpeggiated so as to fall on a pre-selected musical grid580. As described above and as summarized in FIG. 17, a user 500 couplesone or more sensors 508, 510 and/or 515, 517 to the bottom of the user'sfoot/feet or shoe(s) 504/506. As user 500 applies a force to the one ormore sensors 508, 510 and/or 515, 517, force measurements arecommunicated to transceiver unit 110 where the force data may becompiled as a pressure curve showing applied force over time. Computingdevice 116, via its processor and programmed gait analytic algorithm,may then interrogate the pressure curve data to detect event onset,wherein the event data is quantized to a MIDI signal (e.g., MIDI signal540, 542, 544, 546, 548, 550). The MIDI signal may be further calculatedrelative to a preset maximum force setting so as to condition the MIDIsignal as a square wave having a magnitude indicative of the measuredapplied force. The quantized MIDI signal (or calibrated square wave) maythen be modified and arpeggiated onto a pre-selected musical grid 580.

In accordance with an aspect of the present invention, computing device116 may be selectively configured to arpeggiate notes on a sixteenthnote grid. Thus, should an outputted MIDI note be mistimed with respectto the sixteenth note timing, computing device 116, via the gaitanalytic algorithm, may “hold back” or postpone the auditory cueslightly so as to properly place the note on the chromatic scale,temporal grid, or in the correct rhythmic timing. As a result, theauditory biofeedback cues are temporally adjusted so as to avoiddiscordant noise while, instead, producing an auditorily pleasingpattern. Furthermore, the incoming notes can be modified chromaticallyby transposing them to the nearest note in a specified key. Thisarpeggiated pattern may assist the user during therapy as the user is nolonger focused on trying to properly step to an externally-dictated andartificial metronome, but can focus on improving gait mechanics. In afurther example, the tempo of arpeggiation may be selectively adjusted(faster or slower) so as to encourage a change in gait or step rate, forexample, increased or decreased gait velocity.

Turning now to FIG. 18, an exemplary method 600 in accordance with thepresent invention may include: 602) receiving, at the computing device,a series of respective first signals for each of the first and secondsensors from the radio transceiver unit; 604) converting, via thecomputing device, the series of respective first signals into a seriesof respective second signals; 606) quantizing, via the computing device,each respective second signal within the series of respective secondsignals; 608) modifying, via the computing device, the timing of eachrespective second signal to a pre-selected rhythmic or musical grid sothat each respective second signal manifests as a respective real-timeMusical Instrument Digital Interface (MIDI) audio biofeedback cue havinga note length determined as a function of the pulse length of thedigital pulse of the corresponding respective first signal; 610)analyzing, via the computing device, the user's gait as a function ofthe audio biofeedback cues; and 612) adapting, via the computing devicebased upon the gait analysis, the pre-selected temporal or musical gridto adjust entrainment of the user's gait.

In a further aspect of the present invention, step 614 may includecalibrating the sensitivity of the audio biofeedback cue having a notelength determined as a function of the pulse length of the digital pulseof the corresponding respective first signal, while at 616, one or bothsensor regions may be calibrated to the applied force setting by havinga user exert partial or full weight-bearing pressure on each of the oneor both sensors and adjusting the sensitivity of the respective sensorsuch that the full wearing-bearing pressure is set as 100% appliedforce. At 618, each respective second signal may then be calculated as afunction of the 100% applied force setting whereby the quantized secondsignal is displayed as a percentage of applied force as a function ofthe pressure applied to the respective sensor to produce the associatedfirst signal.

While the inventive system and method have been shown and described withreference to certain preferred embodiments thereof, it will beunderstood by those skilled in the art that various changes in form anddetails may be made therein without departing from the spirit and scopeof the invention as described.

What is claimed is:
 1. A method for adapting auditory biofeedback cuesto adjust a user's gait, wherein the user is equipped with a system forcreating a sensory output from the user's physical movements wherein thesystem includes a first sensor configured to be located on or proximateto a heel of a first foot of the user and a second sensor configured tobe located on or proximate to a toe of the first foot of the user,wherein each of the first and second sensors is adapted to detect theuser's movement and generate respective movement data in real-time,wherein the respective movement data includes one or both of pressurewave data and acceleration curve data; a radio transceiver unit having atransceiver and a microprocessor, wherein the microprocessor is coupledto each of the first and second sensors and is programmed with an eventonset detection algorithm wherein the microprocessor is operable todetect an event onset of a respective pressure peak in the generatedrespective movement data, convert the respective movement data into arespective discrete digital pulse having a pulse width and transmit eachrespective digital pulse as a respective first signal; a receiverconfigured to receive each transmitted respective first signal; and acomputing device including a processor programmed with a gait analyticalgorithm configured to convert each respective first signal to arespective second signal, the method comprising: a) receiving, at themicroprocessor, a series of respective first signals for each of thefirst and second sensors; b) converting, via the microprocessor, theseries of respective first signals into a series of respective secondsignals; c) quantizing, via the microprocessor, each respective secondsignal within the series of respective second signals; d) calibrating,via the processor, the sensitivity of the audio biofeedback cue wherebya note length of the audio biofeedback cue is determined as a functionof the pulse width of the digital pulse of the corresponding respectivefirst signal, e) calibrating, via the processor, the sensitivity of oneor both of the first and second sensors such that the gait analyticalgorithm sets full weight-bearing pressure as 100% applied force; f)calculating, via the processor, each respective second signal as apercent weight bearing force as a function of the pressure applied tothe respective sensor relative to the 100% applied force; g) adapting,via the processor, one or both of a volume and a pitch of the audiobiofeedback cue as a function of percent weight bearing force; h)modifying, via the processor, the timing of each respective secondsignal to a pre-selected temporal or musical grid so that eachrespective second signal manifests as a respective real-time MusicalInstrument Digital Interface (MIDI) audio biofeedback cue; i) analyzing,via the processor, the user's gait as a function of the audiobiofeedback cues; j) analyzing, via the processor, a patient state; andk) adapting, via the processor based upon the gait analysis, thepre-selected temporal grid to adjust entrainment of the user's gait tothe tempo of the audio biofeedback cues.
 2. A method for adaptingauditory biofeedback cues to adjust a user's gait, the methodcomprising: a) providing a system for creating a sensory output from auser's physical movements wherein the system includes: i) a first sensorconfigured to be located on or proximate to a heel of a first foot ofthe user and a second sensor configured to be located on or proximate toa toe of the first foot of the user, wherein each of the first andsecond sensors is adapted to detect the user's movement and generaterespective movement data in real-time, wherein the respective movementdata includes one or both of pressure wave data and acceleration curvedata; ii) a radio transceiver unit having a transceiver and amicroprocessor, wherein the microprocessor is coupled to each of thefirst and second sensors and the transceiver is programmed with an eventonset detection algorithm wherein the microprocessor is operable todetect an event onset of a respective pressure peak in the generatedrespective movement data, convert the respective movement data into arespective discrete digital pulse having a pulse width and transmit eachrespective digital pulse and event onset as a respective first signal;iii) a receiver configured to receive each transmitted respective firstsignal; and iv) a computing device having a processor and a gaitanalytic algorithm configured to convert each respective first signal toa respective second signal, b) receiving, at the radio transceiver unit,a series of respective first signals for each of the first and secondsensors; c) quantizing, at the radio transceiver unit, each respectivefirst signal within the series of respective first signals; d)converting, at the computing device, the series of respective firstsignals into a series of respective second signals; e) modifying, at thecomputing device, the timing of each respective second signal to apre-selected temporal grid so that each respective second signalmanifests as a respective real-time Musical Instrument Digital Interface(MIDI) audio biofeedback cue; and f) analyzing, via the gait analyticalgorithm, the user's gait as a function of the audio biofeedback cues.3. The method of claim 2 wherein the radio transceiver unit includes: i)a memory populated with the event onset detection algorithm, and ii) thefirst processor to process the first signals, second signals or audiobiofeedback cues.
 4. The method of claim 2 wherein the series ofrespective first signals is cached within a memory of the computingdevice, wherein the gait analytic algorithm compares the cached firstsignals with the pre-selected temporal grid, and wherein a tempo of thepre-selected temporal grid or MIDI audio biofeedback cue is adaptedbased upon the comparison.
 5. The method of claim 2 wherein the eventonset detection algorithm utilizes an edge detection or a peak detectionprotocol.
 6. The method of claim 2 wherein the first sensor and thesecond sensor are respective first and second sensor regions within awhole-foot sensor.
 7. The method of claim 2 wherein one or both of thefirst sensor and second sensor includes a pressure sensor and aninertial measurement unit.
 8. The method of claim 2 further comprisingthe step of analyzing, via the gait analytic algorithm, a patient state.9. The method of claim 2 further comprising, calibrating sensitivity ofthe audio biofeedback cue on the computing device whereby a note lengthof the audio biofeedback cue is determined as a function of the pulsewidth of the digital pulse of the corresponding respective first signal.10. The method of claim 9 further comprising, calibrating sensitivity ofone or both of the first and second sensors such that the gait analyticalgorithm sets full weight-bearing pressure as 100% applied force whendisplayed on the computing device.
 11. The method of claim 10 furthercomprising, calculating each respective second signal as a percentweight bearing force as a function of the pressure applied to therespective sensor relative to the 100% applied force when displayed onthe computing device.
 12. The method of claim 11 further comprising,adapting the audio biofeedback cue output from the computing device as afunction of percent weight bearing force.
 13. The method of claim 12wherein one or both of a volume and a pitch of the audio biofeedback cueis adapted.
 14. The method of claim 2 wherein the audio biofeedback cueis synchronized with one or more other MIDI clock signals on thecomputing device.
 15. The method of claim 2 further comprising,adapting, via the processor based upon the gait analysis, thepre-selected temporal grid to adjust entrainment of the user's gait. 16.The method of claim 2 wherein the transceiver is a radio transceiverconfigured to generate and transmit the respective first signals as arespective wireless pre-MIDI signal that is converted by the computingdevice into the respective MIDI signal comprising MIDI notes or MIDIcontinuous controller messages.
 17. The method of claim 16 wherein theprocessor converts each respective first signal into the respectivesecond signal at a higher data rate than the MIDI protocol to therebyreduce latency in the output of the output signal.
 18. The method ofclaim 2 wherein the first and second sensors reside on or within a shoe.19. The method of claim 2 wherein the microprocessor further includes aclock function configured to isolate and transmit the respective firstsignals.
 20. The method of claim 2 wherein the microprocessor furtherincludes a multiplexer function to integrate the movement data in thetransmission of the respective first signals.
 21. The method of claim 2wherein each respective real-time MIDI audio biofeedback cue isgenerated, transmitted and converted in less than 50 milliseconds. 22.The method of claim 2 wherein the inertial measurement unit comprisesone or more of of an accelerometer, gyroscope and a magnetometer.